[SPARK-7689] Remove TTL-based metadata cleaning in Spark 2.0 #10534

JoshRosen · 2015-12-31T00:54:56Z

This PR removes spark.cleaner.ttl and the associated TTL-based metadata cleaning code.

Now that we have the ContextCleaner and a timer to trigger periodic GCs, I don't think that spark.cleaner.ttl is necessary anymore. The TTL-based cleaning isn't enabled by default, isn't included in our end-to-end tests, and has been a source of user confusion when it is misconfigured. If the TTL is set too low, data which is still being used may be evicted / deleted, leading to hard to diagnose bugs.

For all of these reasons, I think that we should remove this functionality in Spark 2.0. Additional benefits of doing this include marginally reduced memory usage, since we no longer need to store timetsamps in hashmaps, and a handful fewer threads.

jerryshao · 2015-12-31T03:08:44Z

Good to remove it :). It has been obsolete for a long time, and will potentially lead to some unexpected behaviors especially in Streaming (like block not found).

SparkQA · 2015-12-31T05:10:15Z

Test build #48528 has finished for PR 10534 at commit 98b732a.

This patch fails from timeout after a configured wait of 250m.
This patch merges cleanly.
This patch adds no public classes.

rxin · 2016-01-01T09:09:09Z

LGTM. We should let @tdas take a look at this though.

…eaning

JoshRosen · 2016-01-02T22:05:01Z

streaming/src/test/scala/org/apache/spark/streaming/StreamingContextSuite.scala

@@ -81,6 +81,7 @@ class StreamingContextSuite extends SparkFunSuite with BeforeAndAfter with Timeo

  test("from conf with settings") {
    val myConf = SparkContext.updatedConf(new SparkConf(false), master, appName)
+    // TODO(josh): Update these exmaples to use a different configuration.


SparkQA · 2016-01-02T23:47:30Z

Test build #48592 has finished for PR 10534 at commit e6482fa.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2016-01-02T23:49:34Z

Test build #48590 has finished for PR 10534 at commit 5ffe30f.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2016-01-03T09:47:19Z

Test build #48601 has finished for PR 10534 at commit 1d96791.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds no public classes.

JoshRosen · 2016-01-03T20:50:04Z

Jenkins, retest this please.

gatorsmile · 2016-01-03T20:52:48Z

All the recent test builds failed due to the same issue.

SparkQA · 2016-01-03T22:21:57Z

Test build #48616 has finished for PR 10534 at commit 1d96791.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds no public classes.

JoshRosen · 2016-01-04T08:53:56Z

Jenkins, retest this please.

SparkQA · 2016-01-04T13:18:26Z

Test build #48645 has finished for PR 10534 at commit 1d96791.

This patch fails from timeout after a configured wait of 250m.
This patch merges cleanly.
This patch adds no public classes.

andrewor14 · 2016-01-04T18:39:12Z

+1

tdas · 2016-01-04T21:44:02Z

core/src/main/scala/org/apache/spark/storage/BlockManager.scala

@@ -75,7 +77,7 @@ private[spark] class BlockManager(

  val diskBlockManager = new DiskBlockManager(this, conf)

-  private val blockInfo = new TimeStampedHashMap[BlockId, BlockInfo]
+  private val blockInfo = new ConcurrentHashMap[BlockId, BlockInfo]


Isnt it simpler to keep the Scala map interface? Will minimize changes in rest of code in this class.

I was worried about the pitfalls of .getOrElseUpdate not being atomic on ConcurrentHashMaps that had been wrapped into Scala maps.

Good point.

SparkQA · 2016-01-04T23:52:29Z

Test build #48693 has finished for PR 10534 at commit 0868fab.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds no public classes.

…eaning

tdas · 2016-01-06T23:28:26Z

docs/configuration.md

@@ -1439,7 +1428,7 @@ Apart from these, the following properties are also available, and may be useful
        <td>
            A comma separated list of ciphers. The specified ciphers must be supported by JVM.
            The reference list of protocols one can find on
-            <a href="https://blogs.oracle.com/java-platform-group/entry/diagnosing_tls_ssl_and_https">this</a>
+            a href="https://blogs.oracle.com/java-platform-group/entry/diagnosing_tls_ssl_and_https">this</a>


why is this < removed??

Whoops, thanks for catching that.

tdas · 2016-01-06T23:29:11Z

LGTM, pending tests.

SparkQA · 2016-01-07T00:03:45Z

Test build #48878 has finished for PR 10534 at commit 9704bc2.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

JoshRosen · 2016-01-07T02:33:26Z

Jenkins, retest this please.

SparkQA · 2016-01-07T04:27:41Z

Test build #48889 has finished for PR 10534 at commit 2451963.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

rxin · 2016-01-07T04:50:19Z

Merging. Thanks.

JoshRosen · 2020-04-17T21:30:25Z

Cross-references to related PRs (to aid other code archaeologists):

Earlier discussion of deprecating this flag: [SPARK-7689] Deprecate spark.cleaner.ttl #6220
The PR which introduced periodic forced driver GCs: [SPARK-8414] Ensure context cleaner periodic cleanups #10070

JoshRosen added 5 commits December 30, 2015 16:07

Remove MapOutputTracker cleaner.

9427635

Remove from HttpBroadcast

23669a7

Remove from BlockManager.

f2c2f5d

Delete TimeStampedHashSet

3940e97

All of the rest of the changes.

98b732a

JoshRosen changed the title ~~[SPARK-7689][WIP] Remove TTL-based metadata cleaning~~ [SPARK-7689][WIP] Remove TTL-based metadata cleaning in Spark 2.0 Dec 31, 2015

Merge remote-tracking branch 'origin/master' into remove-ttl-based-cl…

5ffe30f

…eaning

JoshRosen reviewed Jan 2, 2016
View reviewed changes

Thread-safety fixes.

e6482fa

Update StreamingContextSuite tests.

1d96791

JoshRosen changed the title ~~[SPARK-7689][WIP] Remove TTL-based metadata cleaning in Spark 2.0~~ [SPARK-7689] Remove TTL-based metadata cleaning in Spark 2.0 Jan 4, 2016

tdas reviewed Jan 4, 2016
View reviewed changes

JoshRosen added 3 commits January 6, 2016 14:26

Merge remote-tracking branch 'origin/master' into remove-ttl-based-cl…

08b026e

…eaning

Update tests to use dummy time config

75ac218

Remove old todo

8b7ca1c

JoshRosen force-pushed the remove-ttl-based-cleaning branch from 0868fab to 8b7ca1c Compare January 6, 2016 22:30

Add missing import.

9704bc2

tdas reviewed Jan 6, 2016
View reviewed changes

Update configuration.md

2451963

asfgit closed this in 8e19c76 Jan 7, 2016

JoshRosen deleted the remove-ttl-based-cleaning branch August 29, 2016 19:21

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[SPARK-7689] Remove TTL-based metadata cleaning in Spark 2.0 #10534

[SPARK-7689] Remove TTL-based metadata cleaning in Spark 2.0 #10534

JoshRosen commented Dec 31, 2015

jerryshao commented Dec 31, 2015

SparkQA commented Dec 31, 2015

rxin commented Jan 1, 2016

JoshRosen Jan 2, 2016

SparkQA commented Jan 2, 2016

SparkQA commented Jan 2, 2016

SparkQA commented Jan 3, 2016

JoshRosen commented Jan 3, 2016

gatorsmile commented Jan 3, 2016

SparkQA commented Jan 3, 2016

JoshRosen commented Jan 4, 2016

SparkQA commented Jan 4, 2016

andrewor14 commented Jan 4, 2016

tdas Jan 4, 2016

JoshRosen Jan 4, 2016

tdas Jan 6, 2016

SparkQA commented Jan 4, 2016

tdas Jan 6, 2016

JoshRosen Jan 6, 2016

tdas commented Jan 6, 2016

SparkQA commented Jan 7, 2016

JoshRosen commented Jan 7, 2016

SparkQA commented Jan 7, 2016

rxin commented Jan 7, 2016

JoshRosen commented Apr 17, 2020

[SPARK-7689] Remove TTL-based metadata cleaning in Spark 2.0 #10534

[SPARK-7689] Remove TTL-based metadata cleaning in Spark 2.0 #10534

Conversation

JoshRosen commented Dec 31, 2015

jerryshao commented Dec 31, 2015

SparkQA commented Dec 31, 2015

rxin commented Jan 1, 2016

JoshRosen Jan 2, 2016

Choose a reason for hiding this comment

SparkQA commented Jan 2, 2016

SparkQA commented Jan 2, 2016

SparkQA commented Jan 3, 2016

JoshRosen commented Jan 3, 2016

gatorsmile commented Jan 3, 2016

SparkQA commented Jan 3, 2016

JoshRosen commented Jan 4, 2016

SparkQA commented Jan 4, 2016

andrewor14 commented Jan 4, 2016

tdas Jan 4, 2016

Choose a reason for hiding this comment

JoshRosen Jan 4, 2016

Choose a reason for hiding this comment

tdas Jan 6, 2016

Choose a reason for hiding this comment

SparkQA commented Jan 4, 2016

tdas Jan 6, 2016

Choose a reason for hiding this comment

JoshRosen Jan 6, 2016

Choose a reason for hiding this comment

tdas commented Jan 6, 2016

SparkQA commented Jan 7, 2016

JoshRosen commented Jan 7, 2016

SparkQA commented Jan 7, 2016

rxin commented Jan 7, 2016

JoshRosen commented Apr 17, 2020